Skip to content

Store API, orchestration fields, and per-state concurrency#1

Merged
DonPrus merged 27 commits intomainfrom
feat/orchestration
Mar 14, 2026
Merged

Store API, orchestration fields, and per-state concurrency#1
DonPrus merged 27 commits intomainfrom
feat/orchestration

Conversation

@DonPrus
Copy link
Contributor

@DonPrus DonPrus commented Mar 14, 2026

Summary

Adds namespaced key-value store with full-text search, orchestration integration fields, and per-state concurrency control for task claiming.

Store API

  • Namespaced KV store: hierarchical namespace + key organization
  • CRUD operations: PUT (upsert preserving created_at), GET single, GET list by namespace, DELETE single, DELETE namespace
  • FTS5 full-text search: GET /store/search?q=... with optional namespace filter
  • FTS5 sync triggers: automatic index updates on insert/update/delete

Orchestration Integration

  • run_id column on tasks — links tasks to workflow runs
  • workflow_state_json column — stores workflow state on tasks
  • Methods: updateTaskRunId, updateTaskWorkflowState

Per-State Concurrency

  • Claim endpoint accepts concurrency.per_state limits
  • countLeasedTasksInState checks current load before claiming
  • Tasks at state limit are skipped, next eligible task returned

Cleanup

  • Removed legacy gate_results system (table, queries, API handlers, types)
  • Gate check removed from transition logic
  • E2E tests updated: removed gate_results tests, added Store API coverage

Infrastructure

  • MIT license
  • SQLite FTS5 extension enabled in vendored build
  • OpenAPI spec updated with Store endpoints and per_state concurrency
  • CI: tests on Ubuntu, macOS, Windows
  • Release builds for 7 targets
  • Branch protection on main

DonPrus added 15 commits March 13, 2026 19:25
Adds a namespace-based KV store (store table) with full CRUD via REST
endpoints (PUT/GET/DELETE /store/{namespace}/{key}), plus per-state
concurrency limits on lease claims.
- Apply migration 002 (run_id, workflow_state_json columns; drop gate_results)
  in store.zig migrate() which was previously skipped
- Add run_id and workflow_state_json columns to 001_init.sql for fresh databases
- Remove gate_results table from 001_init.sql (no longer used)
- Remove dead store functions: listTasks, listEvents, listArtifacts
  (replaced by paginated variants)
- Remove all gate_results references from openapi.json, README, docs
- Add store (KV) endpoints and /tasks/{id}/run-state to openapi.json
- Add store schemas (StorePutRequest, StoreEntry) to openapi.json
- Add workflow_id field to PipelineStateDef in openapi.json
- Update README API surface table with store endpoints
- Update docs/api.md, architecture.md, workflows.md to remove gates,
  add store section
- Update AGENTS.md and README project layout with missing modules
  (types.zig, config.zig, migration files)
INSERT OR REPLACE deleted and re-inserted the row, resetting
created_at_ms on every update. Switch to INSERT ... ON CONFLICT DO
UPDATE so only value_json and updated_at_ms change. Add test assertion
to verify created_at_ms is preserved across upserts.
The response structs in types.zig are unused — api.zig builds JSON
responses via manual writer calls instead of serializing these types.
The orchestration migration (run_id, workflow_state_json columns and
gate_results drop) is inlined in store.zig's migrate() function. The
.sql file on disk was never loaded via @embedfile and served no purpose.
The claim handler accepts an optional concurrency object with a
per_state map for per-state concurrency limits, but this was missing
from the OpenAPI spec.
Remove types.zig entry and add clarification to store.zig description.
…etry tests

Cover the core claim-transition workflow, per-state concurrency limits
blocking excess claims, role-based claim filtering, and fail-with-retry
policy behavior. All tests use :memory: SQLite and std.testing.allocator.
handleGetTaskRunState freed the store-allocated run_id string using the
request arena allocator (whose free is a no-op), causing the GPA
allocation to leak. Use ctx.store.freeOwnedString instead.
AGENTS.md referenced src/types.zig which was removed. Added
export_manifest.zig and from_json.zig to both AGENTS.md and README.md
project layout sections.
gate_results table was dropped in the orchestration migration. The e2e
test still referenced POST/GET /runs/{id}/gates endpoints and expected
transition to fail when required_gates were unsatisfied. These endpoints
no longer exist and gates are now informational only.

Added comprehensive Store API e2e tests covering put, get, upsert,
list namespace, FTS5 search, delete key, and delete namespace.
Add -DSQLITE_ENABLE_FTS5 compile flag so migration 004 (Store FTS5
virtual table) works on all platforms.
@DonPrus DonPrus changed the title Feat/orchestration Store API, orchestration fields, and per-state concurrency Mar 14, 2026
DonPrus added 12 commits March 14, 2026 07:08
- Add sanitizeFts5Query() that wraps each search term in double quotes
  with internal quote escaping, preventing malformed FTS5 syntax from
  causing 500 errors
- Reject empty/whitespace-only queries with 400 instead of passing to SQLite
- Reject "search" as a namespace in handleStorePut with 400 error
- Document the /store/search route reservation in routing comments
Decode %XX sequences and '+' as space in search query parameter.
Fixes E2E test where q=endpoints+methods was searched as literal
string with '+' instead of two separate words.
@DonPrus DonPrus merged commit 6349dca into main Mar 14, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant